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Abstract. An approach due to Wojtkovski [13], based on the Jacobi fields, is applied to 
study sets of 3-period orbits in billiards on hyperbolic plane and on two-dimensional sphere. 
It is found that the set of 3-period orbits in billiards on hyperbolic plane, as in the planar 
case, has zero measure. For the sphere, a new proof of Baryshnikov's theorem is obtained 
which states that 3-period orbits can form a set of positive measure if and only if a certain 
a natural condition on the orbit length is satisfied. 



1. Introduction 

This article provides a unified approach, based on the Jacobi fields, to study open sets of 
3-period orbits in billiards on manifolds with constant curvature. Specifically, we consider 
spherical and hyperbolic cases. While the spherical as well as Euclidean case has been 
treated previously, our result for the billiards on hyperbolic plane is a new one. Billiards on 
manifolds of constant curvature have been studied earlier in [7], [H]. In [H], billiard domains 
on the sphere, containing open sets of periodic orbits, have been constructed. In P] and in 
this article, the main goal is to assure that except for those special cases, 3-period orbits 
have zero measure. 

The billiard system on a two dimensional Riemannian manifold (M, g) consists of the do- 
main Q with a piecewise smooth boundary dQ and a mass point moving along the geodesies 
inside the domain. Whenever the mass hits the boundary, it refiects according to Fermat's 
principle so as to extremize the path length. That leads to the familiar law: the angle of 
incidence is equal to the angle of refiection. 

Periodic orbits are a natural object of study in dynamical systems. One important question 

concerns the presence of large sets, in particular sets of positive measure, of periodic orbits 

in the billiard ball problem. Informally speaking, measure corresponds to the probability 

that a given orbit is periodic. This question has been originally motivated by spectral 

geometry problems. The second term of the Weyl asymptotics for the Dirichlet problem 

in a bounded domain has a certain special form if periodic orbits of the associated billiard 

problem have zero measure [13], see also a survey article by Gutkin There is a natural 

invariant measure for the billiard map which can be defined as follows: let s be an arclength 

parameter coordinatizing the boundary and let G [0, it] be the angle of the outcoming 

ray from the boundary measured in the counterclockwise direction. The billiard ball map 
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T : [dQ X [0, vr]] — )■ [dQ x [0, tt]] which takes an outcoming ray to another one obtained after 
reflection from the boundary, preserves the measure /i = sin (p d(j) ds , see e.g. [3]. 

Our motivation to study the structure of the set of periodic orbits in non-Euchdean ge- 
ometries is that this understanding may help one with the planar case for higher period 
orbits. It is also expected that eigenvalue asymptotics in non-Euclidean geometries would 
also require understanding the structure of the sets of periodic orbits. 

For the planar billiard problem, it is easy to see that two period orbits have zero measure, 
since these orbits must be normal to the boundary at both ends. Similarly, this is the case 
for a billiard on IP. On the other hand, a billiard on §^ with boundary given by equator 
has two-period orbits of positive measure. This has to do with the presence of conjugated 
points on 

For the period 3, the problem on existence of positive measure sets is already non-trivial. 
The first result on zero measure of 3-period orbits in planar billiards was obtained by Rychlik, 
see [H], relying on symbolic calculations, which were later removed in pjj. Using Jacobi 
fields, Wojtkovski gave an elegant simple proof of Rychlik's theorem. Subsequently, there 
have been extensions to other types of billiard systems: higher dimensional ([15]), outer 
bilhards ([SI HO]), and spherical ([2]). Recently, a proof for the period 4 case has been 
announced by Glutsyuk and Kudryashov [H]. 

Our main result is 

Theorem 1. The set of 3-period orbits in any billiard on has zero measure. 

In order to prove this theorem, we extend the Jacobi fields approach from [13] and present 
the unified proof which treats all three billiard systems on the constant curvature manifolds 
in the same manner. Our argument proceeds independently of the underlying geometry until 
we get the compatibility condition. Then, using the relevant cosine formula, which depends 
on the geometry, we obtain the relation that must be satisfied on a neighborhood filled with 
3-period orbits 

kg{so) = sm%^o)F{L), 

where L is the length of 3-period orbits, kg{so) is geodesic curvature at one of the vertices, 
(po is the angle of the billiard orbit with the tangent to the boundary at this vertex and sq 
is the value of the arclength parameter s at the vertex. The function F{L) depends on the 
underlying Riemannian manifold 

f I on E2 

F{L) = I coth (f ) on H2 
[cot(f) on §2. 

From this formula it is possible to classify sets of 3-period orbits. In particular, we obtain 
a new proof of a theorem by Baryshnikov on the spherical case [2] where sub-Riemannian 
geometry methods were used. 
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Theorem 2. Let P3 be the set of 3-period orbits in the billiard domain dQ on Assume 
that some orbit {xo,Xi,X2) G P3 has perimeter L = tt, Svr or Stt and that some arcs of dQ 
containing Xq,Xi,X2 belong to great circles. Then {xq,Xi,X2) G int{P^) and P3 has positive 
measure. Otherwise, (xq, Xi, 0:2) ^ int{P'i). In particular, if none of 3-period orbits are of 
the above special type, then P3 has an empty interior and is the set of zero measure. 

Remark 1. We only prove that the set of 3-period orbits on (and on §^ when the special 
condition is not satisfied) has an empty interior. The stronger statement about zero measure 
follows verbatim the argument in [13], page 163. 



2. Billiard system on the surface of constant curvature 

2.1. Jacobi fields. Let Q be a smooth domain on a surface of constant curvature x. The 
biUiard ball inside Q travels along the geodesies and reflects at the boundary. Let 7(e, r) be 
a one-parameter family of geodesies where |e| < eo,— 00 < r < 00. 

For the reader's convenience, we briefly recall the derivation of the Jacobi fields, see e.g. 
[1] or [T]. The Jacobi field is defined by 

and it satisfies the Jacobi equation 

where V denotes the covariant derivative and R is the curvature tensor. As usual, we are 
interested in the component of the Jacobi field, that is perpendicular to 7. Therefore, it can 
be expressed as 

J(r) = J(r)n(r), 

where J(r) is a scalar function and n(r) is a unit vector field perpendicular to 7. If the surface 
has constant curvature x, then one obtains a scalar equation with constant coefficients 

(1) J"(r) + xJ(r) = 0. 

According to the standard result in the theory of differential equations, the solution of the 
Jacobi equation is uniquely defined if two initial conditions J(0) and J'(0) are given. 

2.2. Evolution and refiection matrices. Consider billiards on the hyperbolic plane 
and the 2-sphere §^ which have the curvature x = — 1 and x = 1 respectively. Solving the 
Jacobi equation ([T]), we obtain 

U \ - I "^(^^ cosh(r) + J'(0) sinh(r) on W 
~ 1 J(0) cos(r) + J'(0) sin(r) on S\ 
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In each case, we obtain the evolution matrix P{t) 



where -P(r) = < 



J(r) 
J'{r) 




cosh(r) 
sinh(r) 



cos r 



sin(r) 



( AO) 



cos(r) 



sm T 



sinh(r) 
cosh(r) 



) 




which describes the changes of the Jacobi field over time. 

Note that the corresponding evolution matrix in the Euclidean case is given by 




When the billiard ball hits the boundary at x = (s, 0), the Jacobi field is transformed by 
the linear map R{x) which is essentially the same as the refiection map in the Euclidean case 



Note that the refiection matrix is directly related to the classical mirror formula, see [13]. 

One should expect that the refiection matrix R{x) for the billiard on a two dimensional 
Riemannian manifold should have the same form as in the Euclidean case [13]. Neverthe- 
less, we provide some justification. Consider a one-parameter family of geodesies 7(e, r) 
refiecting from the billiard boundary dQ on a two dimensional Riemannian manifold. In 
an e-neighborhood of the refiection point xq of 7(0, tq), the manifold can be represented as 
a smooth two dimensional surface in M^. Projecting the geodesies and the boundary onto 
the tangent plane at Xq, we obtain the corresponding structure on the plane: a family of 
orbits refiecting from the boundary. It is easy to estimate that the angles as well as dis- 
tances before and after the projection, differ by O(e^). This is mainly due to the expansion 
cose = 1 — -|- O(e^). Also, straightforward estimates show that the projected boundary 
curve will have the curvature equal to the geodesic curvature of dQ with the accuracy O(e^). 
As quadratic terms do not affect linear transformations, the reflection map will have the 
same form as in the Euclidean case with k replaced by kg. 




where R{x) 
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Figure 1. Periodic orbits with period 3 in a billiard on EP 



3. Billiard on the hyperbolic plane 

In this section we prove Theorem [T] Assume that there is an open set of 3-period orbits. 
Then we must have and DT^ equal to the identity, which implies 



(2) 



P{n)Rixi)P{To)R{xo)P{T2)R{x2) 



where / is the identity map, Xo,Xi and X2 are the collision points, and tq,T2,ti, are the 
distances between collision points, see Figure [TJ This relation can be also rewritten as 



(3) 



Pin)R{x,)P{ro) = R-\x2)P-\r2)R-\xo 



which takes the form 



cosh(ri) sinh(ri) 
sinh(ri) cosh(ri^ 



-1 

sin(flii) 



cosh(ro) sinh(ro) 
sinh(ro) cosh(ro) 



-1 



2) 



cosh(r2) 
- sinh(r2) 



- sinh(r2) 
cosh(r2) 



-1 

2fcg(a:o) 
sm(<^o) 
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After simplification, we equate the top right components to gelj^ 

2kg{xi) sinh(ro) sinh(ri) 



(4) sinh(ro + n) - sinh(r2) 



sml(/)i , 



We define 9 to be the interior angle between two adjacent segments of an orbit, that is, 
6 = Ti — 20, see Figure [1} 

Then we alter the hyperbolic cosine formula into 

cosh(r2) = cosh(ro + Ti) — sinh(ro) sinh(ri) — sinh(ro) sinh(ri) cos(6'i). 

We use the half angle formula to get 

(5) cosh(ro + Ti) — cosh(r2) = 2cos^ (^"^) sinh(ro) sinh(ri). 

Combining Q and ([s]), we arrive at 

cosh(ro + Ti) - cosh(r2) sinh(ro + n) - sinh(r2) cos (y) 



cos 



' (I) ^.(^i) 



Note that the length of an orbit L = tq + ri + r2 is invariant. Therefore, 



K{x,) = ^inh(ro + n) - sinh(r2) ^^^3 f^r 



cosh(ro + Ti) — cosh(r2) \ 2 

sinh(L — T2) — sinh(r2) o / 61 

cos ' 



cosh(L — T2) — cosh(r2) \ 2 
cos — coth 



2 J V 2 

= sin^(0i) coth ^— 

This relation must hold for all nearby orbits. In particular, for all orbits starting at 
the same point on the boundary with different angles of reflection. Thus, we obtain a 
contradiction because the right-hand side of the equation is not constant in any interval. 
Therefore, the set of 3-period orbits has an empty interior. Next, following an argument in 
[13] we obtain that the set has zero measure, which ends the proof of the Theorem [l} 



^Compare with the corresponding formula in the Euchdean case: tq + ti — r2 = ^ ^Anlcjn) ' "which was 
derived in |13) . 
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4. Billiard on the 2-sphere 

Now we prove Theorem |2] using the same method. Assuming there is an open set of 3- 
period orbits on we again obtain that and DT^ are equal to the identity. Therefore, 
using ([3]) again 

P(ri)i?(xi)P(ro) = R-\x2)P-\r2)R-\xo) 

we get 

2kg{xi) sin(ro) sin(ri) 



(6) 



sin(ro) cos(ri) + cos(ro) sin(ri) — sin(r2) 



sm(0i. 



Note that this relation is the same as ^ if trigonometric functions are replaced with their 
hyperbolic counterparts. 

Combining ^ and the modified version of spherical cosine formula, we arrive at 

(7) kg{xi) = sin^(0i)cot ' 




Figure 2. The special case L = ir. The billiard boundary, which is an octant, 
is formed by three mutually orthogonal great circles lo,h,h- The billiard orbit 
{xo,xi,X2) is obtained from the half of the great circle {xo,Xq) by reflecting 
from the boundaries. 

If cot (^) 7^ 0, then we have the same contradiction as in the hyperbolic case. When 
L = (2n + 1)tt, we have cot (^) = and kg = 0. In this case, there could exist open sets of 
3-period orbits. 
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Now, we discuss the characteristics of the bilhards on which open sets of 3-period orbits 
exist. Note that only orbits without repetition are considered, and this assumption hmits 
our cases to L = 7r,37r, or Svr, see also [H]. 

Proposition 1. Consider a spherical triangle Xq,Xi^X2 on the unit sphere with perimeter 
L = TCjSTi or Svr. Let lo,h,h be the great circles passing through the vertices orthogonal to 
the corresponding bisectors. Then, these great circles intersect at the right angles and any 
billiard boundary containing segments oflo,h,l2 passing through xo,xi,X2 will have an open 
set of 3-period orbits. 

Proof. Let Iq be a geodesic on and xq be any point on Iq. Create two geodesies, h and I2, 
that are perpendicular to Iq and to each other, but do not pass through xq- Consider any 
geodesic segment, r, of length vr, whose endpoint is xq. Denote the angle between r and Iq as 
a. Through two reflections over li and I2, this line r forms a triangle of length vr within the 
boundary created by Iq, li, and 12- Since Xq and a were arbitrary, any 3-period orbit of length 
71 must be contained in one octant, which is formed by Iq, li, and I2, whose intersections 
are orthogonal. In particular, this implies that all orbits in the octant are 3-periodic except 
those which hit the corners. 

Consider three great circles that intersect at Xq, Xi, and X2. The total length of the lines is 
Qtt. This implies that an orbit of length Stt is the complement of AxoXiX2- It follows that 
an orbit of length Stt must have vertices on Iq, li, I2 as in the vr case. 

Now we consider a 3-period orbit of length 37r. Since it is impossible to create an orbit where 
To = Ti = T2 = TT, we look at the two other possible cases; when < tq < vr, vr < ri, r2 < 27r, 
and < To, Ti < TT, TT < r2 < 27r. In CclSG 1, clS shown in Figure |3| we know that /\xqXix'2 has 
perimeter vr and that X2 is antipodal to Xg. This implies that X2 lies on I2. Note that case 2 
is simply the complement of case 1. Therefore, we conclude that a 3-period orbit of length 
37r also has vertices on /q, ^i, and l2- □ 

The last proposition completely classifies the special cases when open sets of 3-period 
orbits occur. If a given 3-period orbit has perimeter L 7^ vr, 37r, Svr then the relation ([t]) 
implies that this orbit has an empty interior in P3. If L = vr, 37r or Svr but for some vertex Xi 
the geodesic curvature kg{s) does not vanish identically on any open boundary arc containing 
then again leads to the same contradiction. 

Finally applying the argument in [T3J, we obtain that if the special cases do not occur the 
set of 3-period orbits has zero measure. This ends the proof of Theorem |2] 
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